Improving usability benchmarking for the eHealth domain:The development of the eHealth UsaBility Benchmarking instrument (HUBBI)

BACKGROUND: Currently, most usability benchmarking tools used within the eHealth domain are based on re-classifications of old usability frameworks or generic usability surveys. This makes them outdated and not well suited for the eHealth domain. Recently, a new ontology of usability factors was developed for the eHealth domain. It consists of eight categories: Basic System Performance (BSP), Task-Technology Fit (TTF), Accessibility (ACC), Interface Design (ID), Navigation & Structure (NS), Information & Terminology (IT), Guidance & Support (GS) and Satisfaction (SAT). OBJECTIVE: The goal of this study is to develop a new usability benchmarking tool for eHealth, the eHealth UsaBility Benchmarking Instrument (HUBBI), that is based on a new ontology of usability factors for eHealth. METHODS: First, a large item pool was generated containing 66 items. Then, an online usability test was conducted, using the case study of a Dutch website for general health advice. Participants had to perform three tasks on the website, after which they completed the HUBBI. Using Partial Least Squares Structural Equation Modelling (PLS-SEM), we identified the items that assess each factor best and that, together, make up the HUBBI. RESULTS: A total of 148 persons participated. Our selection of items resulted in a shortened version of the HUBBI, containing 18 items. The category Accessibility is not included in the final version, due to the wide range of eHealth services and their heterogeneous populations. This creates a constantly different role of Accessibility, which is a problem for a uniform benchmarking tool., CONCLUSIONS: The HUBBI is a new and comprehensive usability benchmarking tool for the eHealth domain. It assesses usability on seven domains (BSP, TTF, ID, NS, IT, GS, SAT) in which a score per domain is generated. This can help eHealth developers to quickly determine which areas of the eHealth system’s usability need to be optimized

Contextual Health Information Behavior in the Daily Lives of People with Type 2 Diabetes:A Diary Study in Scotland

Changes in lifestyle can have positive effects on treating type 2 diabetes (T2D), like sporting or healthy eating. Therefore, a person diagnosed with T2D is often advised to make healthy choices throughout the day, in addition to other interventions such as medication. To do this, he or she needs health information to support decision-making. Literature describes ample categorizations of types of (health) information behavior and theoretical models that explain the factors that drive people to search for, encounter or avoid information. However, there are few longitudinal studies about triggers and factors in daily life that affect health information behavior (HIB). This study was set up to identify triggers, actions and outcomes for active, passive and avoidant HIB situations in daily life among Scots with Type 2 diabetes (T2D) to identify points of attention for communication strategies. Twelve participants took part in a four-week diary study. Every day, participants received an online diary form to describe active, passive or avoidant HIB situations. Data collection resulted in 53 active, 120 passive and 25 avoidant diary entries. Seven active HIB contexts (e.g., experiencing symptoms, cooking dinner, sports training) and five passive HIB contexts (e.g., home, work, medical facility) were identified. Four motivations for avoidance were found (e.g., time constraints, no health trigger). These results can be used to supplement the theoretical models of health information behavior. Furthermore, health professionals can use these results to support their clients with T2D in the self-management of their health, by guiding them to trustworthy sources of health information and lowering barriers for searching health information

Meet my HUBBI: he's an expert on eHealth usability

Establishing proper usability is like finding the right person to marry. You want a person that is compatible with you. Someone who acknowledges your needs, you enjoy being around with, you easily get along with, and ‘fits’ within your world of people you hold dear. With usability, it is not that much different. The system should be compatible with its intended users by addressing their needs, making sure that they like to use the system, making it easy to use the system and fitting the system within the intended use context. During its 40 years of existence in the field of human-computer interaction, the concept of usability has been narrowed down, broadened and evaluated with qualitative and quantitative methods. This was all done to better specify the usability concept and create instruments to identify usability issues and evaluate usability of technology. But this process has not been aligned with technological innovations. The result? Usability has become an important concept to consider during development processes of technology, including eHealth. However, little attention has been given to updating the concept of usability to (1) accommodate the new innovative possibilities of modern-day technologies, and (2) to the manner in which we nowadays use technology – it is not an addition to the things we do, but has become an integral part of our live in the ways we work, relax, sport, communicate, receive care or manage our health. In this thesis, I focused on the latter two: how to (1) include usability within the development process of eHealth and (2) improve usability benchmarking for eHealth services. Within the second part, I describe the development and validation of a new usability benchmarking tool for eHealth: the eHealth UsaBility Benchmarking Instrument (HUBBI)

Assessing usability of eHealth technology: A comparison of usability benchmarking instruments

Background: It is generally assumed that usability benchmarking instruments are technology agnostic. The same methods for usability evaluations are used for digital commercial, educational, governmental and healthcare systems. However, eHealth technologies have unique characteristics. They need to support patients’ health, provide treatment or monitor progress. Little research is done on the effectiveness of different benchmarks (qualitative and quantitative) within the eHealth context. Objectives: In this study, we compared three usability benchmarking instruments (logging task performance, think aloud and the SUS, the System Usability Scale) to assess which metric is most indicative of usability in an eHealth technology. Also, we analyzed how these outcome variables (task completion, system usability score, serious and critical usability issues) interacted with the acceptance factors Perceived benefits, Usefulness and Intention to use. Methods: A usability evaluation protocol was set up that incorporated all three benchmarking methods. This protocol was deployed among 36 Dutch participants and across three different eHealth technologies: a gamified application for older adults (N = 19), an online tele-rehabilitation portal for healthcare professionals (N = 9), and a mobile health app for adolescents (N = 8). Results: The main finding was that task completion, compared to the SUS, had stronger correlations with usability benchmarks. Also, serious and critical issues were stronger correlated to task metrics than the SUS. With regard to acceptance factors, there were no significant differences between the three usability benchmarking instruments. Conclusions: With this study, we took a first step in examining how to improve usability evaluations for eHealth. The results show that listing usability issues from think aloud protocols remains one of the most effective tools to explain the usability for eHealth. Using the SUS as a stand-alone usability metric for eHealth is not recommended. Preferably, the SUS should be combined with task metrics, especially task completion. We recommend to develop a usability benchmarking instrument specifically for eHealth

Time to act mature—Gearing eHealth evaluations towards technology readiness levels

It is challenging to design a proper eHealth evaluation. In our opinion, the evaluation of eHealth should be a continuous process, wherein increasingly mature versions of the technology are put to the test. In this article, we present a model for continuous eHealth evaluation, geared towards technology maturity. Technology maturity can be determined best via Technology Readiness Levels, of which there are nine, divided into three phases: the research, development, and deployment phases. For each phase, we list and discuss applicable activities and outcomes on the end-user, clinical, and societal front. Instead of focusing on a single perspective, we recommend to blend the end-user, health and societal perspective. With this article we aim to contribute to the methodological debate on how to create the optimal eHealth evaluation design

Conceptualizing usability for the ehealth context: Content analysis of usability problems of ehealth applications

Background: Usability tests can be either formative (where the aim is to detect usability problems) or summative (where the aim is to benchmark usability). There are ample formative methods that consider user characteristics and contexts (ie, cognitive walkthroughs, interviews, and verbal protocols). This is especially valuable for eHealth applications, as health conditions can influence user-system interactions. However, most summative usability tests do not consider eHealth-specific factors that could potentially affect the usability of a system. One of the reasons for this is the lack of fine-grained frameworks or models of usability factors that are unique to the eHealth domain. Objective: In this study, we aim to develop an ontology of usability problems, specifically for eHealth applications, with patients as primary end users. Methods: We analyzed 8 data sets containing the results of 8 formative usability tests for eHealth applications. These data sets contained 400 usability problems that could be used for analysis. Both inductive and deductive coding were used to create an ontology from 6 data sets, and 2 data sets were used to validate the framework by assessing the intercoder agreement. Results: We identified 8 main categories of usability factors, including basic system performance, task-technology fit, accessibility, interface design, navigation and structure, information and terminology, guidance and support, and satisfaction. These 8 categories contained a total of 21 factors: 14 general usability factors and 7 eHealth-specific factors. Cohen κ was calculated for 2 data sets on both the category and factor levels, and all Cohen κ values were between 0.62 and 0.67, which is acceptable. Descriptive analysis revealed that approximately 69.5% (278/400) of the usability problems can be considered as general usability factors and 30.5% (122/400) as eHealth-specific usability factors. Conclusions: Our ontology provides a detailed overview of the usability factors for eHealth applications. Current usability benchmarking instruments include only a subset of the factors that emerged from our study and are therefore not fully suited for summative evaluations of eHealth applications. Our findings support the development of new usability benchmarking tools for the eHealth domain

Tailoring persuasive electronic health strategies for older adults on the basis of personal motivation: Web-based survey study

Background: Persuasive design, in which the aim is to change attitudes and behaviors by means of technology, is an important aspect of electronic health (eHealth) design. However, selecting the right persuasive feature for an individual is a delicate task and is likely to depend on individual characteristics. Personalization of the persuasive strategy in an eHealth intervention therefore seems to be a promising approach. Objective: This study aimed to develop a method that allows us to model motivation in older adults with respect to leading a healthy life and a strategy for personalizing the persuasive strategy of an eHealth intervention, based on this user model. Methods: We deployed a Web-based survey among older adults (aged >60 years) in the Netherlands. In the first part, we administered an adapted version of the revised Sports Motivation Scale (SMS-II) as input for the user models. Then, we provided each participant with a selection of 5 randomly chosen mock-ups (out of a total of 11), each depicting a different persuasive strategy. After showing each strategy, we asked participants how much they appreciated it. The survey was concluded by addressing demographics. Results: A total of 212 older adults completed the Web-based survey, with a mean age of 68.35 years (SD 5.27 years). Of 212 adults, 45.3% were males (96/212) and 54.7% were female (116/212). Factor analysis did not allow us to replicate the 5-factor structure for motivation, as targeted by the SMS-II. Instead, a 3-factor structure emerged with a total explained variance of 62.79%. These 3 factors are intrinsic motivation, acting to derive satisfaction from the behavior itself (5 items; Cronbach alpha=.90); external regulation, acting because of externally controlled rewards or punishments (4 items; Cronbach alpha=.83); and a-motivation, a situation where there is a lack of intention to act (2 items; r=0.50; P<.001). Persuasive strategies were appreciated differently, depending on the type of personal motivation. In some cases, demographics played a role. Conclusions: The personal type of motivation of older adults (intrinsic, externally regulated, and/or a-motivation), combined with their educational level or living situation, affects an individual’s like or dislike for a persuasive eHealth feature. We provide a practical approach for profiling older adults as well as an overview of which persuasive features should or should not be provided to each profile. Future research should take into account the coexistence of multiple types of motivation within an individual and the presence of a-motivation

Embodied conversational agent appearance for health assessment of older adults: Explorative study

Background: Embodied conversational agents (ECAs) have great potential for health apps but are rarely investigated as part of such apps. To promote the uptake of health apps, we need to understand how the design of ECAs can influence the preferences, motivation, and behavior of users. Objective: This is one of the first studies that investigates how the appearance of an ECA implemented within a health app affects users' likeliness of following agent advice, their perception of agent characteristics, and their feeling of rapport. In addition, we assessed usability and intention to use. Methods: The ECA was implemented within a frailty assessment app in which three health questionnaires were translated into agent dialogues. In a within-subject experiment, questionnaire dialogues were randomly offered by a young female agent or an older male agent. Participants were asked to think aloud during interaction. Afterward, they rated the likeliness of following the agent's advice, agent characteristics, rapport, usability, and intention to use and participated in a semistructured interview. Results: A total of 20 older adults (72.2 [SD 3.5] years) participated. The older male agent was perceived as more authoritative than the young female agent (P=.03), but no other differences were found. The app scored high on usability (median 6.1) and intention to use (median 6.0). Participants indicated they did not see an added value of the agent to the health app. Conclusions: Agent age and gender little influence users' impressions after short interaction but remain important at first glance to lower the threshold to interact with the agent. Thus, it is important to take the design of ECAs into account when implementing them into health apps